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ABSTRACT 

In  support  of  the  Army’s  objective  of  developing  embedded  training  for  the  Future  Combat  System  (FCS),  Army 
Research,  Development  &  Engineering  Command  (RDECOM)  sponsored  the  development  of  a  technology 
demonstration  and  experiment  with  an  integrated  architecture  linking  intelligent  evaluation  mechanisms  with  their 
Command  and  Control  Vehicle  (C2V)  testbed.  Automated  evaluation  methods  based  on  Intelligent  Tutoring 
Systems  (ITS)  techniques  are  applied  to  monitor  simulation  and  testbed  events,  and  deliver  feedback  during  scenario 
execution  via  messages  published  to  the  simulation  environment.  The  logic  for  these  intelligent  evaluations  is 
captured  in  hierarchical  agent  behaviors,  and  indexed  to  training  principles  identified  by  subject  matter  experts. 
This  paper  summarizes  the  findings  from  this  effort,  including  technical  methods  as  well  as  the  results  of 
experiments  with  human  test  subjects  to  measure  the  effectiveness  of  the  system  for  training.  Specifically,  the 
improvement  in  performance  over  time  among  test  subjects  receiving  automated  feedback  was  contrasted  with 
subjects  receiving  other  forms  of  instruction  such  as  an  instructor-led  after  action  review.  These  results  provide  a 
basis  for  the  discussion  of  the  way  forward  with  FCS  embedded  training.  Ultimately,  with  the  application  of 
automated  training  methods  such  as  embedded  ITS  and  structured  training,  a  major  potential  benefit  is  the  ability  to 
train  in  settings  where  human  instructors  are  not  present  or  available.  The  work  building  the  demonstration  system 
sheds  light  on  several  areas  of  potential  future  work  in  support  of  developing  full  scale  Intelligent  Structured 
Training  systems  to  realize  these  benefits.  In  addition  to  further  validation  of  the  approach,  significant 
developmental  areas  include  integration  and  compatibility  with  simulation  common  components  like  OneSAF  that 
are  likely  to  be  used  by  the  FCS  program,  scenario  authoring  tools  to  streamline  the  process  for  subject  matter 
experts,  and  rapid  behavior  definition  methods  for  simulated  opposing  forces  to  heighten  realism  and  enhance 
training  benefits. 
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INTRODUCTION 

In  support  of  the  Army’s  objective  of  developing 
embedded  training  for  the  Future  Combat  System 
(FCS),  the  Army  Research,  Development  & 
Engineering  Command  (RDECOM)  Simulation  & 
Training  Technology  Center  (STTC)  Embedded 
Combined  Anns  Team  Training  and  Mission  Rehearsal 
(ECATT-MR)  Army  Technology  Objective  (ATO) 
sponsored  the  development  of  a  technology 
demonstration  and  experiment  with  an  integrated 
architecture  linking  intelligent  evaluation  mechanisms 
with  their  Command  and  Control  Vehicle  (C2V) 
testbed.  This  testbed  is  based  on  likely  embedded 
virtual  simulation  common  components  like  OneSAF 
and  Terrain  Databases  built  for  virtual  simulations  with 
the  focus  being  to  examine  the  feasibility  of  developing 
an  Intelligent  Structured  Training  System  that  would 
operate  in  these  classical  virtual  simulation 
environments  as  opposed  to  environments  created 
solely  for  the  purpose  of  intelligent  tutoring 
applications.  Automated  evaluation  methods  based  on 
Intelligent  Tutoring  Systems  (ITS)  techniques  are 
applied  to  monitor  simulation  and  testbed  events,  and 
deliver  feedback  during  scenario  execution  via 
messages  published  to  the  simulation  environment. 
The  logic  for  these  intelligent  evaluations  is  captured  in 
hierarchical  agent  behaviors,  and  indexed  to  training 
principles  identified  by  subject  matter  experts.  This 
paper  summarizes  the  findings  from  this  effort, 
including  technical  methods  as  well  as  the  results  of 
experiments  with  human  test  subjects  to  measure  the 
effectiveness  of  the  system  for  training. 

Specifically,  the  improvement  in  performance  over 
time  among  test  subjects  receiving  automated  feedback 
was  contrasted  with  subjects  receiving  other  forms  of 
instruction  such  as  an  instructor-led  after  action 
review.  These  results  provide  a  basis  for  the  discussion 
of  the  way  forward  with  FCS  embedded  training. 
Ultimately  with  the  application  of  automated  training 
methods  such  as  embedded  ITS  and  structured  training, 
a  major  potential  benefit  is  the  ability  to  train  in 
settings  where  human  instructors  are  not  present  or 
available.  The  work  building  the  demonstration  system 
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sheds  light  on  several  areas  of  potential  future  work  in 
support  of  developing  full  scale  Intelligent  Structured 
Training  systems  to  realize  these  benefits.  In  addition 
to  further  validation  of  the  approach,  significant 
developmental  areas  include  integration  and 
compatibility  with  simulation  engines  that  are  likely  to 
be  used  by  the  FCS  program,  scenario  authoring  tools 
to  streamline  the  process  for  subject  matter  experts,  and 
rapid  behavior  definition  methods  for  simulated 
opposing  forces  to  heighten  realism  and  enhance 
training  benefits. 


OVERVIEW  OF  DOMAIN  AND  APPLICATION 

FCS  has  identified  training  to  be  a  Key  Performance 
Parameter  (KPP).  The  approach  to  providing  training 
has  been  identified  to  be  a  fully  embedded  approach, 
allowing  Soldiers  to  train  and  maintain  tactical 
knowledge  proficiency  on  the  same  devices  they  take 
to  war.  Ideally  this  coidd  be  performed  with  no 
appended  or  training  unique  equipment,  thus  making 
the  training  and  operational  modes  interchangeable 
based  on  unit  functions.  Since  the  training  facilities  of 
the  present  will  be  replaced  by  troops  training  in  motor 
pools  and  doing  mission  rehearsal  in  assembly  areas, 
the  need  to  provide  robust  training  packages  becomes 
much  more  important. 

Higher  level  cognitive  training  requires  that  the 
trainee’s  performance  be  evaluated  and  that  he  receives 
appropriate  feedback  on  that  performance.  However, 
in  the  Embedded  Training  context,  where  most  FCS 
training  is  planned  to  occur,  instructors  will  not  be 
available  to  provide  these  functions  like  they  would  at 
a  fixed  training  site.  Intelligent  software  is  required  to 
monitor  a  trainee’s  actions  in  a  simulated  scenario, 
evaluate  those  actions  against  Army  Tactics, 
Techniques,  and  Procedures,  and  provide  real-time, 
dynamic,  one-on-one  feedback  using  the  Soldier 
machine  interface,  audio,  and  other  mechanisms,  as 
appropriate.  The  field  of  Intelligent  Tutoring  Systems 
provides  techniques  to  accomplish  these  tasks  in 
automated  fashion.  A  key  design  objective  in  the 
application  of  ITS  technology  ultimately  in  FCS 
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embedded  training  is  the  ability  to  interface  with  and 
utilize  likely  virtual  simulation  common  components 
such  as  OneSAF,  and  operate  with  the  virtual  databases 
supported  by  current  simulations. 

To  explore  the  use  of  ITS  technologies  in  this  context, 
a  task  area  under  the  Army’s  FCS  concept  was  selected 
for  the  implementation  of  an  Intelligent  Structured 
Trainer  to  be  applied  in  a  set  of  experiments  with 
students  to  measure  training  results.  Within  this  task 
area,  the  FCS  Soldier  manning  a  Command  and 
Control  Vehicle  (C2V)  crewstation  must  remotely 
control  robotic  platforms  to  perform  reconnaissance 
and  engage  the  enemy.  The  operator  must  maintain 
situational  awareness  and  apply  tactical  decision¬ 
making  principles  in  a  heightened  information-rich 
setting  with  distributed  vehicles  and  sensors  under  his 
command.  The  increased  flexibility  allows  him 
multiple  ways  to  detect,  track,  and  identify  targets. 
The  following  figures  show  some  of  the  primary 
screens  in  the  Soldier  interface  in  the  C2V  crewstation. 


Figure  1.  Unmanned  Ground  Vehicle  (UGV) 
Gunner  Sensor  View 


Figure  2.  C2V  Robotic  Asset  Mission  Status  Tool 


Figure  3.  Operational  Control  Unit 

The  ITS  technology  applied  for  this  project  centers 
around  automated  evaluation  and  feedback  on  operator 
performance,  resulting  in  a  training  methodology 
referred  to  as  Intelligent  Structured  Training.  Example 
feedback  is  shown  in  the  following  screen  from  the 
crewstation. 


You  have  failed  to  submit  a 
SITREP.  The  Correct  Procedure 


is:(1)  Select  Report,  (2)  Select 
Location.  (3)  Select  Teleop  (4) 
Select  Route,  (5)  Select  Cancel. 


Figure  4.  Immediate  Directive  Feedback  Presented 
in  the  Visualization  Screen 


An  Intelligent  Structured  Trainer  automates  the  process 
of  monitoring  student  actions  and  providing  feedback, 
either  as  Immediate  Directive  Feedback  (IDF)  in  real¬ 
time  or  as  delayed  feedback  to  be  incorporated  with 
After  Action  Review  (AAR).  With  practice  in  a 
simulated  crewstation  environment,  students  can  open 
a  scenario,  go  through  an  exercise,  and  receive 
customized  feedback  on  their  application  of  the  FCS 
concept  of  operations.  The  training  application 
developed  for  this  experiment  is  not  a  full  Intelligent 
Tutoring  System  in  the  traditional  sense,  as  it  does  not 
incorporate  other  technologies  from  the  realm  of  ITSs 
such  as  instructional  planning  and  persistent  student 
models,  but  the  term  “ITS”  is  used  as  a  referential  term 
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in  this  paper  for  the  Intelligent  Structured  Trainer 
developed  incorporating  ITS  technology. 

The  process  of  constructing  the  scenario  for  this 
experiment  started  with  a  task  analysis  highlighting  the 
functional  objectives  for  each  system  (Command  and 
Control  Vehicle  [C2V],  Unmanned  Aerial  Vehicle 
[UAV],  Unmanned  Ground  Vehicle  [UGV],  and 
Armed  Reconnaissance  Vehicle  -Assault  [ARV-A], 
etc),  which  guided  the  process  of  task  development. 
The  desired  outcome  was  the  training  of  the  robotics 
operator  of  the  FCS  Company  Level  C2V  within  a 
Combined  Arms  Unit  of  Action  (UA).  This  was  a 
subordinate  task  embedded  within  the  larger  task  of 
training  of  the  FCS  C2V  crew,  with  a  focus  on  the 
robotics  operator.  Therefore,  the  primary  training 
audience  was  the  Robotics  NCO  and  his  interaction 
with  the  company  Executive  Officer  (XO),  as  well  as 
the  Driver  and  Vehicle  Commander,  whose  secondary 
duties  include  the  operation  of  robotic  vehicles  within 
the  company. 


Figure  5.  Partial  Task  Decomposition 


Task  development  included  the  identification  of  a 
proposed  mission  (Route  Reconnaissance)  and  a 
decomposition  of  tasks  needed  to  support  this  goal. 
This  task  decomposition  process  included  the 
identification  of  conditions  under  which  the  tasks 
would  be  accomplished  and  the  development  of 
measures  of  performance  and  effectiveness.  Using  a 
cognitive  task  analysis  approach,  a  modified  Goals, 
Operators,  Methods,  and  Selection  (GOMS)  Rules 
Model  was  developed,  including  frizzy  production  and 
selection  rules.  This  GOMS  model  enabled  the 
identification  of  goals  and  sub-goals  supporting  the 
overall  tactical  mission  of  Conduct  a  Route 
Reconnaissance.  This  critical  step  allowed  for  the 
development  of  subordinate  tasks,  conditions  and 
standards  for  small  unit  collective  tasks,  as  well  as 
identifying  the  skills,  knowledge  and  attributes 
required  by  the  robotics  operator.  Figure  5  below 
shows  a  partial  task  decomposition. 


The  diagram  above  shows  how  the  tactical  mission. 
Conduct  a  Route  Reconnaissance,  can  be  decomposed 
into  its  major  supporting  tasks.  The  first  major  subgoal 
was  the  Route  Recon  task  itself.  This  task  immediately 
triggered  a  concurrent  task,  Submit  Reports.  The 
Submit  Report  Sub  Task  is  a  recurrent  task  which  is 
triggered  anytime  there  is  a  change  state  (beginning 
movement,  launching  an  Unmanned  Vehicle,  engaging 
the  enemy,  etc).  The  Route  Recon  task  required  two 
major  sub  tasks;  Review  the  Mission  and  selection  of  a 
Method  of  Observation.  Method  of  Observation 


included  five  principal  techniques  (variables)  and  each 
of  these  techniques  were  defined  and  refined,  one 
technique  at  a  time.  This  task  refinement  yielded 
conditions  and  standards  for  each  technique.  After  all 
five  techniques  were  defined  separately  they  were 
again  refined  to  work  in  conjunction  with  one  another. 
This  entire  process  yielded  Tasks,  Conditions,  and 
Standards  for  each  task  and  subtask.  It  also  identified 
tasks  and  actions  that  other  crew  members  were 
required  to  perform  to  support  the  Robotics  Operator, 
and  additional  conditions  for  the  execution  of  the 
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training  scenario.  For  example,  once  a  target  was 
detected,  the  Robotics  NCO  was  required  to  select  an 
engagement  type,  generally  consisting  of  a  subset  of 
tasks  within  the  Cooperative  Engagement  technique. 
Permission  to  engage  targets  would  generally  flow 
from  the  Commanding  Officer  to  the  XO.  Requests  for 
engagement  authority  and  permission  to  engage  were 
simplified  into  a  single  action.  Call  for  Fire,  and 
automatically  handled  within  the  simulation.  This 
identified  a  needed  precondition  for  the  training 
scenario,  authority  to  engage,  and  an  area  for  future 
experimentation,  the  automated  handling  of  calls  for 
fire. 

In  order  to  appropriately  evaluate  for  operator 
performance  during  an  exercise,  the  training  system 
requires  two  primary  sets  of  data.  First,  real-time 
information  about  the  state  of  the  simulated  exercise  is 
necessary  in  order  to  assess  the  conditions  under  which 
the  student  performs  different  actions.  Data  in  this 
category  includes  information  about  vehicle  locations, 
headings,  control  status  and  sensor  input,  as  well  as  the 
outcomes  of  contact  with  enemy  forces.  Outcomes  can 
provide  useful  feedback  to  the  student  as  to  the 
appropriateness  of  decisions  during  an  exercise. 
Flowever,  simulation  outcomes  are  not  always  relevant 
to  student  feedback,  as  a  free-play  simulation  allows 
for  a  degree  of  flexibility  where  negative  outcomes 
may  occur  even  if  the  student  has  performed  well  on  all 
or  most  all  the  relevant  principles.  The  second 
category  of  data  involves  the  student’s  activities  within 
the  software  and  user  interfaces.  This  second  category 
is  particularly  helpful  in  diagnosing  mistakes  that  the 
student  has  made,  since  it  reveals  useful  information 
about  the  student’s  intentions. 


AUTOMATED  EVALUATION  METHODOLOGY 

The  automated  evaluations  at  the  core  of  the  ITS  were 
implemented  as  Behavior  Transition  Networks  (BTNs), 
an  extension  of  a  Finite  State  Machine  (FSM)  approach 
suited  to  the  complexity  of  the  operational  domain, 
involving  free  action  in  complex,  dynamic  scenarios. 
With  this  approach,  evaluation  BTNs  are  indexed  to 
relevant  principles  through  a  scenario  definition 
interface.  This  is  a  generalizable  task,  which  can  easily 
be  delegated  to  an  instructor  or  subject  matter  expert 


for  training  systems  with  a  plurality  of  exercise 
scenarios.  Thus,  when  the  exercise  is  executed  in  the 
simulation,  each  evaluation  BTN  for  each  principle 
relevant  to  the  scenario  is  activated. 

The  use  of  BTNs  represents  a  number  of  enhancements 
to  the  traditional  FSM  model.  In  particular,  the 
evaluation  machines  are  composed  using  an  authoring 
and  execution  tool  called  SimBionic  (Fu  et  al,  2003). 
SimBionic  provides  a  visual  toolset  for  defining 
simulation  behaviors,  with  support  for  multiple  chained 
conditions  between  states,  local  and  global  variables, 
and  the  organization  of  different  BTNs  in  a 
hierarchically  nested  structure.  This  tool  can  be  used 
to  control  the  behavior  of  simulation  entities,  or  in 
instructional  applications,  to  specify  the  behavior  of 
instructional  agents  running  as  virtual  entities  within  a 
simulation.  The  run-time  engine  for  these  BTNs 
interprets  all  active  behaviors  in  parallel  and  invokes 
actions  and  predicates  implemented  within  the 
application  as  specified  by  each  behavior.  A  behavior 
may  be  composed  of  actions,  conditions,  connectors, 
and  other  behaviors.  Actions  are  functions  that  are 
invoked  to  carry  out  activities,  such  as  sending  a 
message;  these  often  correspond  to  functionality 
provided  in  the  API  of  a  given  simulation  or  other 
environment  with  which  the  behaviors  are  integrated. 
Conditions  perform  checks,  typically  whether  an  entity 
is  in  a  specified  state,  invoking  predicates  to  determine 
state  information  from  the  simulation  or  testbed. 
Predicates  are  functions  that  return  values  representing 
the  state  information  of  an  entity,  such  as 
IsTheTargetLazed(target).  This  is  a  predicate  that 
returns  true  in  the  C2V  testbed  if  the  user  has 
performed  the  laze  function  on  a  target  or  location. 
Connectors  connect  actions,  conditions  and  behaviors 
to  specify  their  relative  order  of  execution. 

These  BTNs  form  the  structure  for  defining  and 
executing  automated  evaluations  that  monitor  states 
and  performance  in  the  integrated  testbed.  For  each 
entity  created  in  the  testbed,  the  ITS  creates  a 
corresponding  monitoring  entity  which  invokes 
specified  evaluation  behaviors,  and  places  it  under  the 
control  of  the  behavior  execution  run-time  engine.  As 
a  general  rule,  behaviors  are  invoked  for  each  principle 
in  the  principle  hierarchy  which  applies  for  the  training 
domain  and  testbed. 
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Figure  6.  Example  Evaluation  BTN,  “Follow  Correct  Engagement  Procedure” 


An  example  ITS  behavior  is  shown  in  Figure  6  above, 
taken  directly  from  the  SimBionic  authoring 
environment.  This  behavior  monitors  how  the  operator 
performs  the  procedure  for  engaging  a  target.  Actions 
and  sub-behaviors  are  represented  in  rectangles,  and 
conditions  are  represented  in  ovals.  For  each  step  of 
the  proper  engagement  procedure,  this  BTN  checks  if 
the  step  has  been  performed  according  to  the  proper 
sequence.  Due  to  the  sequential  nature  of  the  correctly 
executed  engagement  procedure,  this  behavior  serves 
as  an  example  where  it  is  appropriate  to  capture  several 
specific  feedback  points  corresponding  to  different 
training  sub-principles.  For  example,  if  the  user 
performs  a  call  for  fire  on  a  target  without  having  lazed 
the  target,  this  violates  one  of  the  procedural  execution 
principles.  As  a  result,  the  behavior  executes  an  action, 
SendMessage(message2),  where  message2  is  indexed 
to  specific  feedback  text  authored  in  a  central  feedback 
collection.  The  SendMessage  function  adds  the 
message  to  the  message  queue  to  be  delivered  and 
presented  to  the  operator  in  the  testbed  interface,  as 
Immediate  Directive  Feedback.  This  interaction  is  then 
recorded  via  the  RecordEngagement  action. 

With  the  BTN-based  approach  for  implementing 
automated  evaluation  measures,  these  evaluations 


function  like  instructional  agents,  identifying 
conditions  that  are  instructionally  significant.  Since 
the  goal  is  to  mimic  tasks  performed  by  a  human 
instructor,  these  evaluation  measures  also  logically 
serve  as  a  tool  for  automated  instructional  decision 
making  about  triggering  OPFOR  responses  to  certain 
conditions  or  simulation  states.  As  a  result,  a 
mechanism  was  developed  for  controlling  simulation 
entities  in  the  OneSAF  Testbed  Baseline  (OTB) 
remotely  from  the  instructional  evaluation  measures. 
This  form  of  responsive  OPFOR  behavior  was 
implemented  with  specific  operational  retreat 
behaviors.  Still  more  complex  OPFOR  behavior 
remains  an  area  for  potential  further  development,  in 
conjunction  with  possible  future  investigation  with  the 
more  flexible  design  of  the  OneSAF  Objective  System 
(OOS)  simulation. 


EXPERIMENT  DESIGN 

Test  subjects  were  given  a  combination  of  human 
tutored  and  computer  aided  instructions  while 
occupying  the  Robotics  NCO  crew  station  in  a  FCS 
equipped  C2V  Testbed  simulator,  located  at  STTC. 
This  experiment  required  test  subjects  to  learn  both 
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Procedural  and  Conceptual  Knowledge  tasks  in  order 
to  accomplish  a  tactical  mission.  Procedural 
Knowledge  tasks  included  the  control  of  unmanned 
robotic  assets  in  a  virtual  environment  and  learning  the 
correct  procedures  for  conducting:  reconnaissance, 
surveillance  and  target  acquisition  tasks,  two  target 
engagement  techniques,  and  submitting  situation 
reports.  Conceptual  Knowledge  tasks  included  learning 
a  defined  set  of  the  tactical  principles  associated  with 
the  planning  and  execution  of  a  tactical  reconnaissance 
mission  and  their  associated  supporting  tasks.  Test 
Subjects  were  given  a  timed  tactical  scenario  which 
required  them  to  perform  a  reconnaissance  mission  in 
order  to  demonstrate  proficiency  in  the  application  of 
both  Procedural  and  Conceptual  Knowledge  (Sanders, 
2005). 

The  available  sample  for  this  experiment  included 
undergraduate  students  consisting  of  14  males  and  11 
females,  ranging  in  age  from  18  to  33  with  11  subjects 
pursuing  a  Bachelor  of  Arts  degree  and  9  subjects 
pursuing  a  Bachelor  of  Science  degree.  The  average 
test  subject  was  a  20.5  year  old  male  pursuing  a 
Bachelor  of  Arts  degree  and  was  self-assessed  as 
having  good  computer  experience  with  several  types  of 
software  programs.  Ten  test  subjects  were  randomly 
assigned  to  one  of  two  feedback  condition  groups: 
Immediate  Directive  Feedback  (IDF)  only  and  AAR 
Only  (Delayed  Feedback). 

In  the  Intelligent  Structured  Trainer  developed  for  this 
experiment,  every  action  performed  by  test  subjects 
within  the  scenario  is  evaluated  to  determine  if  it  is  the 
correct  response  to  the  current  situation.  If  the  response 
is  incorrect,  immediate  feedback  is  provided  in  the 
form  of  an  error  message  provided  to  the  test  subject. 
No  immediate  feedback  was  provided  for  following 
correct  procedures.  The  three  types  of  immediate 
feedback  included: 

•  Battlefield  heuristic  feedback  -  “Conduct  a  sensor 
scan  before  beginning  movement” 

•  Error  detection  and  directive  feedback  -  “Submit 
a  report  anytime  there  is  a  change  to  the  tactical 
situation” 

•  Directive  feedback  -  “You  have  failed  to  correctly 
submit  a  SITREP.  The  correct  procedure  is. . .” 

Immediate  Directive  Feedback  prompts  were  triggered 
either  when  a  test  subject  conducted  a  procedure 
incorrectly,  or  failed  to  take  an  appropriate  action  after 
receiving  an  error  prompt. 


Fluman  tutoring  on  both  Procedural  Knowledge  and 
Conceptual  Knowledge  used  open-ended  prompts 
during  the  AAR  to  elicit  elaboration  and  self¬ 
explanation  from  the  trainees.  The  AAR  protocol  used 
followed  a  typical  Army  After  Action  Review  format 
as  outlined  in  TC  25-20,  A  Leader’s  Guide  to  After- 
Action  Reviews  (DA,  1993),  and  focused  on  answering 
three  top  level  questions:  What  happened;  Why  it 
happened;  and  Flow  to  fix  it.  It  is  during  the  review  of 
tactical  principles  and  the  “Flow  to  fix  it”  portion  of  the 
AAR  that  the  tutor  focuses  on  Conceptual  Knowledge. 
Conceptual  Knowledge  includes  general  tactical 
principles  and  definitions  of  concepts  which  provide  a 
framework  for  Procedural  Knowledge.  For  example, 
Conceptual  Knowledge  includes  understanding  the 
tactical  principles  of  reconnaissance,  the  various 
methods  for  engaging  a  target,  and  the  purpose  for 
submitting  reports;  Procedural  Knowledge  is  the 
understanding  of  the  correct  sequence  of  steps  to 
actually  engage  a  target. 

During  the  AAR,  test  subjects  were  required  to  review 
both  concepts  and  procedures  and  then  asked  open- 
ended,  content  neutral  prompts  to  elicit  elaboration  and 
feedback.  For  example,  one  measure  of  Conceptual 
Knowledge  required  the  test  subject  to  define  the  term 
“Cooperative  Engagement”  and  give  an  example.  As 
part  of  the  AAR,  the  test  subject  reviewed  the 
definition  of  Cooperative  Engagement  and  was  asked 
the  following  questions: 

•  “Can  you  explain  this  concept  in  simple  terms?” 

•  “Can  you  give  an  example?” 

•  “When  have  you  done  something  like  this?” 

If,  in  answering  these  questions,  the  test  subject 
provided  an  incorrect  answer,  the  tutor  identified  the 
answer  as  incorrect  and  asked  the  test  subject  to  try 
again. 

All  test  subjects  were  administered  an  un-timed  paper 
and  pencil  pre-test  to  establish  a  baseline  of  subject 
knowledge.  Each  subject  knowledge  test  consisted  of 
10  Procedural  Knowledge  questions  and  10  Conceptual 
Knowledge  questions.  Procedural  Knowledge 
questions  included  skill  acquisition  tasks  and  asked  the 
test  subjects  to  write  down  the  steps  to  accomplish  a 
procedural  task,  i.e.  “What  is  the  correct  procedure  for 
Submitting  a  SITREP?”  Conceptual  Knowledge 
questions  included  general  tactical  principles  and 
definitions  of  concepts,  i.e.  “What  are  the  tactical 
principles  for  reconnaissance?”  and  “Define  a  Line  of 
Sight  Engagement”.  Test  subjects  had  to  provide  the 
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answers  to  each  question,  were  instructed  not  to  guess, 
and  to  write  “I  do  not  know”  after  any  question  they 
could  not  answer. 

Test  subjects  occupied  the  Robotics  NCO  crew  station 
in  an  FCS  equipped  C2V  simulator  and  received  an 
orientation  to  the  simulator.  Each  training  trial  began 
with  a  review  of  the  procedures  and  training  tasks  to  be 
accomplished  during  the  training  exercise.  After  this 
review,  each  test  subject  executed  a  30  minute  timed 
training  scenario  that  measured  their  ability  to  correctly 
apply  these  concepts  and  conduct  these  procedures  in  a 
manned  simulator.  At  the  end  of  each  training  trial,  the 
test  subjects  assigned  to  the  IDF  Only  feedback 
condition  group  conducted  a  self-paced  review  of  the 
concepts  and  procedures  and  completed  a  subject 
knowledge  test.  The  IDF  Only  feedback  condition 
group  received  no  delayed  feedback  in  the  form  of  an 
AAR.  Test  subjects  assigned  to  the  AAR  Only 
feedback  condition  group  received  a  human  facilitated 
AAR  and  then  completed  a  subject  knowledge  test. 
Four  tests  were  developed,  Subject  Knowledge  Test 
Alpha,  Bravo,  Charlie,  and  Delta  with  the  same  10 
procedural  and  10  concept  knowledge  questions  from 
the  pre-test  presented  in  a  random  order  for  each  test. 
Each  test  subject  was  randomly  assigned  one  of  four 
different  subject  knowledge  tests  upon  completion  of 
each  training  trial.  Tests  were  not  timed.  All  training 
trial  scenarios  were  identical,  with  no  change  to  the 
tactical  scenario  occurring  between  iterations. 

During  phase  two,  a  randomly  assigned,  un-timed 
subject  knowledge  paper  and  pencil  test  was 
administered  to  measure  retention  of  knowledge.  Test 
subjects  were  then  given  a  transfer  task  which  was 
identical  to  the  previously  learned  concepts  about  the 
task.  Conduct  a  Route  Reconnaissance,  and  included 
the  same  goal,  constraints  and  options  for  completing 
the  tasks,  but  the  terrain  was  different  than  the  terrain 
used  during  the  training  tasks.  Test  subjects  had  to  plan 
a  route  on  a  paper  map  and  identify  concepts  and 
procedures  when  answering  questions  about  the  task. 
Test  subjects  then  re-occupied  the  FCS  C2V  simulator 
and  executed  the  transfer  task  on  a  proctor  provided 
scenario. 


EXPERIMENTAL  RESULTS 

The  overall  results  of  the  experiment  support  the 
hypothesis  that  the  timing  and  type  of  feedback 
received  during  training  does  effect  the  acquisition, 
retention  and  transfer  of  both  Procedural  and 
Conceptual  Knowledge.  Significant  differences  did 
exist  in  individual  measures,  suggesting  that  immediate 
directive  feedback  has  a  significant  effect  in  reducing 


the  number  of  errors  committed  while  acquiring  new 
procedural  skills  during  training  as  well  as  retention  of 
these  procedural  skills.  Also,  delayed  feedback,  in  the 
form  of  an  AAR  which  includes  open-ended  prompts 
to  foster  elaboration,  has  a  significant  effect  on  the 
acquisition,  retention  and  transfer  of  higher  order 
Conceptual  and  Procedural  Knowledge  about  a  task. 

The  results  of  this  experiment  show  that  the  timing  of 
feedback,  immediate  and  delayed,  and  the  type  of 
feedback,  directive  and  explanatory,  has  an  impact  on 
the  acquisition,  retention  and  transfer  of  knowledge. 
Immediate  feedback  to  correct  detected  errors 
promoted  skill  acquisition,  retention  and  transfer  for 
procedures.  Providing  feedback  on  procedures,  whether 
immediate  or  delayed,  resulted  in  improvement,  as 
shown  in  Table  1. 

Table  1.  Retention  Scores  for  Procedural 
Knowledge  Following  Feedback 


M 

SD 

IDF  Only 

3.72 

.92 

AAR  Only 

3.80 

1.21 

However,  providing  Immediate  Directive  Feedback 
significantly  reduced  the  amount  of  procedural  errors 
committed  during  training,  versus  Delayed  Feedback, 
as  shown  in  Table  2. 

Table  2.  Procedural  Errors  Following  Feedback 


M 

SD 

IDF  Only 

28.22 

19.50 

AAR  Only 

42.08 

24.59 

An  analysis  of  variance  (ANOVA)  was  performed  on 
the  data,  following  the  practice  of  using  a  combination 
of  between-subjects  and  within-subjects  variables.  For 
this  analysis,  the  between-subjects  factor  is  Feedback 
Conditions  (IDF  Only  and  AAR  Only),  and  the  within- 
subjects  factor  is  Training  Trials.  The  results  of  the 
ANOVA  found  a  significant  main  effect  for  Feedback 
Conditions  (F(l,  1 8)  =  5.87),  and  Training  Trials 
(F(2,24)  =  13.05),  with  a  p  <  .05.  The  ANOVA  results 
tell  us  that  there  are  significant  differences  in  the 
number  of  IDF  prompts  triggered  by  each  test  subject 
and  that  these  differences  are  explained  by  both  the 
number  of  Training  Trials  (learning  occurring  over 
time)  and  by  Feedback  Condition,  IDF  Only  and  AAR 
Only.  The  significantly  lower  number  of  error  prompts 
triggered  during  the  execution  of  the  training  and 
transfer  scenarios  demonstrates  that  the  IDF  Only 
Feedback  Condition  had  a  significant  effect  on  the 
acquisition  and  transfer  of  the  performance  of 
procedures.  This  supports  earlier  studies  on  the  benefits 
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of  immediate  and  directive  feedback  (Anderson  et  al., 
1995;  Buzhardt  and  Semb,  2002;  Dihoff  et  al.,  2004; 
Guthrie,  1971;  Kulik  and  Kulik,  1998). 

Delayed  feedback,  like  that  used  in  the  AAR  Only 
Feedback  Condition  group,  promoted  retention  of  new 
Conceptual  Knowledge  better  than  the  IDF  Only 
condition,  as  shown  in  Table  3. 

Table  3.  Retention  Scores  for  Conceptual 
Knowledge  Following  Feedback 


M 

SD 

IDF  Only 

3.10 

2.03 

AAR  Only 

5.60 

2.12 

One  of  the  areas  of  suggested  future  investigation 
would  involve  experimentation  with  exercise  pre-briefs 
or  post-briefs  which  offer  Conceptual  Knowledge 
explanation  to  the  Soldier  to  boost  retention.  Providing 
feedback  on  performance,  whether  immediate  or 
delayed,  directive  or  explanatory,  resulted  in 
improvements  in  the  acquisition,  retention  and  transfer 
of  knowledge.  These  results  should  prove  useful  to  the 
Army  as  it  continues  its  development  of  its  ET  strategy 
for  the  Future  Force. 


FUTURE  APPLICATIONS 

The  varying  effects  of  feedback  condition  and  trials  on 
the  performance  measures  selected  to  measure  training 
effectiveness  argue  for  additional  research  and 
experimentation.  Individual  findings  of  significance 
between  feedback  condition  measures  within  trials  and 
the  effect  of  within  feedback  condition  measures 
between  trials  suggest  that  feedback  timing  and  type 
does  have  an  overall  effect.  Significant  differences  did 
exist  in  individual  measures,  suggesting  that  immediate 
error  detection  and  directive  feedback  has  a  significant 
effect  in  reducing  the  number  of  errors  committed 
while  acquiring  new  procedural  skills  during  training 
as  well  as  retention  of  these  procedural  skills.  Also, 
delayed  feedback,  in  the  form  of  an  AAR  which 
includes  open-ended  prompts  to  foster  elaboration,  has 
a  significant  effect  on  the  acquisition,  retention  and 
transfer  of  higher  order  Conceptual  Knowledge  about  a 
task.  Therefore,  the  following  list  offers  a  variety  of 
potential  focuses  for  future  work  supporting  Intelligent 
Structured  Training  for  FCS  embedded  platforms. 

•  Develop  an  automated  exercise  pre -brief  or  post¬ 
brief  capability  to  explain  to  the  Soldier  the 
Conceptual  Knowledge  points  within  the  exercise, 
to  amplify  the  benefits  of  immediate  feedback  for 
Conceptual  Knowledge. 


•  Update  the  capabilities  within  the  FCS  C2V 
simulator  to  include  a  freeze  option  to  allow 
trainees  an  opportunity  to  freeze  the  scenario  while 
attending  to  error  detection  and  directive  feedback. 
Enhance  the  procedural  fidelity  by  preventing 
actions  from  occurring  within  the  STE  if  proper 
procedures  are  not  followed. 

•  Automate  the  AAR  protocol  to  remove  humans 
completely  from  the  feedback  process.  Include 
support  for  trainee  self-elaboration  during  the 
AAR. 

•  Refine  the  feedback  condition  categories  and 
effectiveness  measures  to  reflect  updated 
capabilities  with  the  C2V  simulator.  The  mixed 
feedback  condition  should  focus  immediate 
feedback  on  procedural  errors  and  delayed 
feedback  on  Conceptual  Knowledge.  Extend  the 
length  of  time  for  retention,  and  manipulate  the 
variables  to  determine  if  the  linear  association 
between  highest  levels  of  training  achieved  impact 
retention. 

•  Perform  further  principle  hierarchy  development 
for  automated  evaluation  in  a  broader  range  of 
scenarios  within  FCS  tactical  domains. 

•  Refine  interfaces  to  interoperate  with  FCS 
common  components  such  as  OneSAF. 

•  Abstraction  of  principle  application  conditions,  so 
that  evaluations  can  be  applied  to  different 
scenarios  with  minimal  adaptation. 

•  Domain  specific  methods  for  implementing 
automatic  adaptation  of  evaluations  to  different 
scenarios,  such  as  automatic  line  of  sight 
predicates. 

•  Scenario  authoring  tools  to  accelerate  the 
development  of  scenarios  that  operate  as  intended 
on  a  chosen  testbed. 

•  Experimentation  with  enhanced  modes  of 
feedback  during  exercises. 

•  Enhanced  methods  for  defining  and  controlling 
complex  OPFOR  behavior  in  scenarios. 

•  Capabilities  to  adjust  difficulty  levels,  either 
before  execution  or  dynamically  during  execution 
in  response  to  the  actions  of  the  operator. 

•  Extension  to  team  training  use  cases. 

The  Intelligent  Structured  Training  concept  can  be 
applied  widely  throughout  FCS.  The  use  of  BTN- 
based  evaluation  methods  combined  with  real-time, 
dynamic,  tailored  feedback  represents  a  key  innovation 
with  wide  applicability.  Many  FCS  positions  require 
tactical  decision-making  and  the  use  of  complex 
equipment,  under  new  and  developing  doctrine.  Each 
such  position  requires  a  large  number  of  different  tasks 
to  be  performed.  Thus,  the  Intelligent  Structured 
Training  concept  reasonably  applies  to  a  large  number 
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of  tasks  associated  with  different  positions.  In  fact,  an 
instructional  software  system  that  monitors  the  events 
in  a  team  executed  simulation  actually  has  a  greater 
capability  to  note  all  instiuctionally  relevant  actions 
and  events  for  all  participants  than  a  human  instructor 
or  observer  can.  Thus,  although  human  instructors  will 
always  have  value  in  a  training  regimen,  there  is  the 
potential  for  the  average  commander  augmented  with 
intelligent  structured  training  to  greatly  exceed  the 
performance  of  even  the  best  instructor  without  it,  in  at 
least  this  one  metric.  In  fact,  one  augmented 
commander  should  outperform  a  whole  group  of 
instructors.  This  is  a  powerful  notion  for  widespread 
FCS  adoption,  as  the  benefits  can  be  expressed  in  terms 
of  direct  manpower  savings. 

Other  Intelligent  Tutoring  System  concepts  can  also  be 
brought  to  bear,  such  as  student  modeling  and 
instructional  planning.  By  combining  the  performance 
evaluations  across  scenarios  with  information  about 
what  knowledge  and  skills  were  required  for  correct 
performance,  an  ITS  can  estimate  the  mastery  of  a 
trainee  in  each  skill  and  knowledge  component.  This 
information  can  be  used  to  automatically  select 
scenarios  that  force  the  trainees  to  practice  the  skills 
and  knowledge  in  which  they  are  the  weakest  until  a 
specified  level  of  mastery  is  reached.  Scenarios  can 
also  be  selected  to  make  sure  each  skill  and  knowledge 
component  is  tested  a  specified  minimum  number  of 
times.  Additional  remedial  exercises  can  be  added  for 
trainees  that  have  continued  difficulty  in  specific  areas. 


WAY  FORWARD 

In  addition  to  the  experiments  discussed  in  this  paper 
the  C2V  ITS  system  has  been  reviewed  by  several 
possible  users  including  TARDEC  CAT  ATD,  FCS, 
Common  Gunnery  Architecture  (CGA),  PM  OneSAF 
and  Future  Force  Warrior  (FFW)  programs.  Due  to 
possible  interest  in  this  technology  by  the  Future  Force 
Warrior  program  the  RDECOM  is  planning  to  build  a 
prototype  based  around  the  man  wearable  fully 
immersive  virtual  simulations  that  they  are  developing. 
They  will  also  be  developing  streamlined  authoring 
tools  to  build  robust  exercises  at  a  fast  rate  using 
virtual  simulation  operational  environments  and 
common  components.  This  initial  system  experiment 


supports  the  possibility  that  an  ITS  can  be  developed 
and  effectively  utilized  to  provide  effective  embedded 
virtual  tactical  instruction  without  an  instructor.  The 
future  research  will  refine  this  system  to  make 
embedded  training  a  reality  for  the  Army  of  the  Future. 


REFERENCES 

Anderson,  J.,  A.  T.  Corbett,  K.  R.  Koedinger,  &  R. 
Pelletier  (1995).  Cognitive  tutors:  Lessons  learned. 
Journal  of  the  Learning  Sciences,  4,  167-207. 

Buzhardt,  J.,  &  G.  B.  Semb  (2002).  Item-by-Item 
Versus  End-of-Test  Feedback  in  a  Computer- 
Based  PSI  Course.  Journal  of  Behavioral 
Education  11(2):  89-104. 

Department  of  the  Army  (DA),  (1993).  TC  25-30:  A 
leader’s  guide  to  after-action  reviews. 

Dihoff,  R.  E.,  G.  M.  Brosvic,  M.  L.  Epstein,  &  M.  J. 
Cook  (2004).  Provision  of  feedback  during 
preparation  for  academic  testing:  learning  is 
enhanced  by  immediate  but  not  delayed  feedback. 
The  Psychological  Record  54:  207-231. 

Fu,  D.,  R.  Houlette,  R.  Jensen,  &  O.  Bascara  (2003).  A 
Visual,  Object-Oriented  Approach  to  Simulation 
Behavior  Authoring.  Proceedings  of  the 
Industryflnterservice,  Training,  Simulation  & 
Education  Conference  (I/ITSEC  2003). 

Guthrie,  J.  T.  (1971).  Feedback  and  sentence  learning. 
Journal  of  Verbal  Learning  and  Verbal  Behavior 
10:  23-28. 

Kulik,  J.  A.,  &  C.  C.  Kulik  (1988).  Timing  of  feedback 
and  verbal  learning.  Review  of  Educational 
Research  58(1):  79-97. 

Sanders,  M.  J.  (2005).  The  effect  of  immediate 
feedback  and  after-action  reviews  (AARs)  on 
learning,  retention  and  transfer.  Unpublished 
master’s  thesis,  University  of  Central  Florida, 
Orlando. 


2005  Paper  No.  2189  Page  11  of  11 


